International Education Studies; Vol. 5, No. 5; 2012 
ISSN 1913-9020 E-ISSN 1913-9039 
Published by Canadian Center of Science and Education 


Evaluating the Effectiveness of the Chemistry Education by Using the 

Analytic Hierarchy Process 

Mehmet Yiiksel 1 

1 Yenimahalle Technical and Industrial Vocational High School, Ankara, Turkey 

Correspondence: Mehmet Yiiksel, Yenimahalle Technical and Industrial Vocational High School, Ankara, Turkey. 
Tel: 90-312-315-6274. E-mail: m06yuksel@hotmail.com 

Received: May 18, 2012 Accepted: May 30, 2012 Online Published: August 8, 2012 
doi:10.5539/ies.v5n5p79 URL: http://dx.doi.org/10.5539/ies.v5n5p79 


Abstract 

In this study, an attempt was made to develop a method of measurement and evaluation aimed at overcoming the 
difficulties encountered in the determination of the effectiveness of chemistry education based on the goals of 
chemistry education. An Analytic Hierarchy Process (AHP), which is a multi-criteria decision technique, is used 
in the present study. The proposed method consists of four main steps: (1) Determination of the main goals and 
sub-goals of chemistry education; (2) Formation of the AHP model; (3) Making pairwise comparisons and the 
calculation of local weights; (4) Calculation of global weights and the effectiveness of chemistry education. 
Research findings indicated that it is possible to determine the effectiveness of chemistry education based on the 
goals set through the proposed method. 

Keywords: goals of chemistry education, effectiveness, analytic hierarchy process 

1. Introduction 

In its most general sense, chemistry education aims to enable students to acquire the means to understand, 
explain, predict and control nature. Specifically, chemistry education goals are set to give direction to chemistry 
instruction, the curriculum, instructional materials and evaluations, and to the goals which are to be achieved in 
the future (Gayon, 2010). As the perceptions of chemistry, and developments within it have changed over time, 
so too the goals of chemistry education have changed in parallel with them (Mitchell, 1993). The changes 
encountered have also modified the content and methods of chemistry education. For example, the context-based 
science education approach recently proposed for chemistry education requires changes in the curriculum as well 
as the re-examination of the goals of chemistry education, the renewal of subject content, and new approaches to 
assessment (Coenders, Terlouw, & Dijkstra, 2010). For these reasons, it is impossible to consider the goals of 
chemistry education as fixed. On the contrary, it is seen that goals of chemistry education have a dynamic 
structure. 

Different problems concerning chemistry education are the subject of some research in the literature on the goals 
of chemistry education (Boyer & Tiberghien, 1989; Tamir, 1990; Horak, 1993; Mitchell, 1993; Coenders et ah, 
2010; Gayon, 2010). However, besides the content of goals which give direction to chemistry education, it is 
also important to determine to what extent these goals have been reached. In a traditional context, in the field of 
chemistry, the level of reaching educational goals is measured by the academic achievement level of students, as 
is the case in other fields of science (Muijs, 2006). For this reason, many methods for, and approaches to, the 
assessment and evaluation of academic achievement of students have been developed (Kilmen, Kosterelioglu, & 
Kosterelioglu, 2007; Gelbal & Kelecioglu, 2007; Nazligi^ek & Akarsu, 2008; (/elikkaya, Karakusj, & Demirba§, 
2010; Okur & Azar, 2011). Assessing and evaluating student academic achievement in the education process is 
an important indicator that reveals the extent to which a curriculum's purposes have been achieved ((/elikkaya et 
al., 2010). However it is impossible to say that such a method is adequate. Traditional assessment and evaluation 
methods are insufficient for determining the outcomes aimed at in the curriculum (Gelbal & Kelecioglu, 2007; 
Nazlu^ck & Akarsu, 2008; (/elikkaya et al., 2010). This is because there are many factors concerning students 
that must be evaluated within the framework of the goals of education. Evaluating and assessing them with only 
one technique is generally impossible (Kilmen et al., 2007). However, alternative approaches are developed for 
the assessment and evaluation of learning based on the changes in epistemological theories (Baki & Birgiin, 
2002). However the alternative approaches proposed (Baki &Birgiin, 2002; Nazlif^ek & Akarsu, 2008) are also 
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generally considered to be for the assessment and evaluation of student academic knowledge. 

As well as measuring student academic knowledge, it is important to measure to what extent the goals of the 
curriculums are achieved in order to determine the effectiveness of education. This is because the effectiveness 
of education is also among the purposes of educational evaluation (Baykul, 1992). However, a series of 
difficulties and limitations are encountered in the measurement and evaluation of the effectiveness of chemistry 
education on the basis of the goals of chemistry education. The first problem results from the nature of the 
effectiveness concept. Effectiveness is defined in different contexts in the literature. However, in its most general 
sense, it is defined as the extent of the achievement of the purposes. In other words, it shows to what extent the 
expected accomplishments are achieved or how well the outcomes meet the stated goals (Cowan, 1985; Baykul, 
1992; Muijs, 2006; Yilmaz & Ta§dan, 2006; Rupsiene & Bartuseviciene, 2008). However effectiveness studies 
are sophisticated as they generally cover complex activities (Rupsiene & Bartuseviciene, 2008). For this reason, 
difficulties can be encountered in the measurement of effectiveness. There are also difficulties in converting 
effectiveness into a numerical index as it covers different goals with different features (Baykul, 1992). 

Another problem encountered in determining the effectiveness of chemistry education on the basis of its goals 
results from the excessive number of goals in chemistry education. Numerous and different chemistry education 
goals necessitate the measurement and evaluation of the extent to which these goals are achieved from a holistic 
perspective. This is because evaluating the effectiveness of chemistry education on the basis of each goal 
separately will not lead to a holistic evaluation. In the event where the effectiveness of chemistry education is 
evaluated separately for each goal, only the degree to which the goal taken into consideration is realized can be 
determined. However, the relative importance of each goal may vary. For example, it is not possible to say that 
the goal of “presenting fundamental knowledge” and the goal of “promoting awareness of chemistry-related 
careers” have the same relative importance in chemistry education. A similar situation may be true for other 
goals in chemistry education too. A second problem derives from the fact that the structures of the goals of 
chemistry education are generally qualitative and conceptual. Certain difficulties are encountered in the 
quantitative measurement of the goals of chemistry education in this kind of a structure. For example, it is not 
possible to directly quantitatively measure the goal of “emphasizing environmental concepts and world 
problems” in chemistry education. Accordingly, the structure of the measurement technique must be congruent 
with the structure of the goals. 

In the light of the abovementioned information, the present study aims at developing a method of measurement 
and evaluation which overcomes the difficulties and solves the problems encountered in the process of 
determining the effectiveness of chemistry education based on the goals of chemistry education. The Analytic 
Hierarchy Process (AHP), which is a multi-criteria decision technique, developed by Saaty (1980) was used for 
realizing this, the main purpose of the present study. The reason for the use of this technique is that the AHP 
concept is appropriate for the purposes and nature of this research. It was seen in the literature review that the 
AHP technique was used for the solution of some of the problems encountered in the field of education, though 
in a limited number (Feng, Lu., & Bi, 2004;Yetim, 2004; Begicevic and Divjak, 2007; Klein, 2007; Suryadi, 
2007; Warwick, 2007; Uzoka, 2008; Li, Li, & Zhang.,2009; Begicevic, Divjak, & Hunjak, 2010; Tsinidou, 
Gerogiannis, & Fitsilis, 2010; Ho, & Hsu, 2011). 

The remainder of this paper is organized as follows. The second section presents the method of measurement and 
evaluation proposed in accordance with the purpose of the study. This section describes the AHP technique used 
in the study and the basic steps of the proposed method. The third section of the study presents the application of 
the proposed method. Finally, in the fourth section, conclusions and suggestions are presented. 

2. Method 

This section presents the method proposed for the measurement of the effectiveness of chemistry education. As 
expressed in Section 1, the main technique used in the proposed method is the Analytic Hierarchy Process (AHP). 
AHP, which is a multi-criteria decision-making technique, was developed by Saaty (1980). 

In AHP technique, firstly, the problem is defined, and the scope of the problem is determined. In the next stage, 
the hierarchical structure of the problem is formed. There is a purpose in the first level of the hierarchical 
structure, and there are elements and alternatives in the lower levels. The hierarchical structure in AHP may vary 
according to the complexity of the problem and the number of elements and alternatives it includes. Another 
stage in the AHP technique is the formation of pairwise comparison matrixes according to the hierarchical model 
constituted. Pairwise comparison matrixes are square matrixes in n x n size. The comparison of elements of 
matrixes with one another is made according to 1-9 scale of Saaty (1980) (Table 1). 
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Table 1. Saaty’s 1-9 scale for pair-wise comparisons (Saaty, 1980) 


Comparative judgment 

Intensity of importance 

a; and a, are equally important 

1 

a; is weakly more important than aj 

3 

a; is strongly more important than aj 

5 

a; is demonstrably or very strongly more important than aj 

7 

a; is absolutely more important than aj 

9 

Intermediate values between adjacent scale values 

2, 4, 6, 8 


Accordingly, ay indicates pairwise comparison value of i. element and j. element, aji is comparison value of j. 
element and i. element. In AHP, aji value is obtained via aji = 1/aji equality (Saaty, 1980; Saaty, 1982). The 
appearance of the pairwise comparison values in the matrix is as given in equation (1). 
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In AHP, the pairwise comparison number in each matrix varies by the number (n) of elements of the matrix. 
Accordingly, in a matrix containing (n) elements, n (n-1) /2 pairwise comparisons are made. The diagonal values 
of the matrixes are equal to 1.Following the formation of pairwise comparison matrixes in AHP, priority vectors 
of the matrixes are determined. In AHP, priority vector is obtained via the solution of the (A - X nm I) W = 0 
equality (Saaty, 1980). In this equation, A indicates pairwise comparison matrix, W indicates eigenvector and 
/, mllx indicates the biggest eigenvalue of A matrix. 

According to the assumptions of AHP technique, pairwise comparisons must be consistent. Acceptable 
inconsistency levels suggested by Saaty (1994) are as follows: for 3x3 sizes the matrix is 0.05, for 4x4 size the 
matrix is 0.08, and for all other matrix it is 0.1. If the inconsistency ratio of pairwise comparison matrixes is 
lower than the suggested value, the matrix can be considered consistent. Otherwise, pairwise comparisons are to 
be reviewed and renewed by the decision maker. Consistency ratio (CR) is calculated depending on the 
consistency index (Cl) and the random index (RI) values. Consistency index is determined by using the 
eigenvalue (A max ) as following: CI= (A max -n) / (n-1), where n is the matrix size. Then consistency ratio is 
calculated by dividing Cl to Random index (RI) value (CR = Cl / RI). RI is determined depending on the size of 
matrix (n) as seen in Table 2. The operation of the AHP technique takes place in accordance with the 
abovementioned rules. 


Table 2. Values of random index (RI) (Saaty, 1980) 


n 

1 

2 

3 

4 

5 

6 

7 

8 

9 

RI 

0 

0 

0.58 

0.9 

1.12 

1.24 

1.32 

1.41 

1.45 


The method proposed in this study consists of four main steps: 

Step 1: To determine the main goals and sub-goals of chemistry education. In AHP technique, firstly, the 
elements making up the structure of the problem are determined through an analytical examination. In AHP 
technique, after the elements are determined, they are categorized according to their common features. In the 
present study, the goals of chemistry education were determined and categorized in the first step. The 
determination and categorization of the main goals and sub-goals were in the base of the effectiveness of the 
chemistry education and were made in accordance with the methods of the study by Gayon (2010). 
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Step 2: To form an AHP model in accordance with the purpose of the study. The elements that make up the 
problem according to the assumptions of the AHP technique are converted into a hierarchical model in 
accordance with the purpose of the study. In the present study, an AHP model was constituted according to the 
categorization set in the first step. 

Step 3: To make pairwise comparisons of the matrix of each of the main goals and sub-main goals with a 1-9 
scale, and to calculate the local weights and consistency ratio (CR). In AHP technique, evaluations start with 
pairwise comparisons. In this study, too, pairwise comparisons were made according to the 1-9 scale proposed 
by Saaty (1980) by taking into account the constituted AHP model. Another process at this stage is the 
calculation of the local weights of each goal. Local weight indicates the relative importance of each goal within 
the group it exists in. In this stage, lastly, the consistency ratio of pairwise comparisons was calculated. 

Step 4: To calculate the global weights for the sub-goals of chemistry education and the effectiveness of 
chemistry education. In this stage, firstly, the global weights related to sub-goals were calculated. Global weight 
indicates the relative importance of each goal within the whole. In the AHP technique, the value of the whole is 
equal to 1. The final process in this step is the determination of the effectiveness of the chemistry education. As 
stated in the literature review, effectiveness can be explained in different ways. In this study, what is meant by 
the concept of effectiveness is as follows: The degree to which goals anticipated in the curricula are actually 
performed. In the present study, firstly, the expected realization level of the goals was calculated by means of the 
AHP technique. In the second stage, the actual realization level was calculated by taking into account the relative 
weights of goals within all of the goals. In the present study, the expected theoretical value range of the 
effectiveness is 0 - 1. Nearness of effectiveness value to 1 means the perfect realization level of goals, and 
nearness of effectiveness value to 0 means an insufficient realization level of goals. 

3. Results 

In this section, application results related to the proposed method are presented. Results related to the application 
were given according to four steps covered by the method proposed in the second section. The application of the 
proposed method was performed within the scope of the 9th grade chemistry course of public high schools in 
Turkey. 

Step 1: Determination of the main and sub-main goals of the chemistry education. 

In this step of the study, goals and sub-goals taken as a basis for measuring the effectiveness of chemistry 
education were determined. As expressed in the literature review, there are many goals intended to be realized 
through chemistry education. These goals may vary according to progress in the science of chemistry, societal 
perception, and the knowledge level of society. The curriculum and goals of chemistry education in Turkey are 
set by the Ministry of National Education (2007). It may be seen that the curriculum in Turkey aims at the at the 
acquisition of knowledge appropriate to present levels of knowledge. According to the actual curriculum in 
Turkey, the main goal of chemistry education is not simply for the students to gain a specific knowledge of 
chemistry. There are also other goals covered by the curriculum, these concern an understanding of different 
domains in life such as the role of the individual, society, the environment, the economy and technology. 

In the literature review, two main empirical studies (Mitchell, 1993; Gayon, 2010) examining and classifying the 
goals of chemistry education were encountered. The classification made by Gayon (2010) was used in this study 
as it was in accordance with the goals of the actual curriculum for the secondary education chemistry course in 
Turkey. Gayon (2010) grouped the goals of chemistiy education according to the five major categories of science 
education goals. These are scientific knowledge, scientific methods, societal issues, personal needs, and career 
awareness. These five main goals consist of 30 sub-goals. The coding devised for the constitution of the AHP 
model in this study was based on Gayon’s (2010) classification of main goals and sub-goals and is as follows: 

Scientific knowledge (SK) 

• Reflecting the nature and structure of chemistry (SKI) 

• Presenting chemistry as a body of scientific knowledge (SK2) 

• Being interdisciplinary in nature (SK3) 

• Presenting fundamental knowledge (SK4) 

• Developing fundamental understanding of chemical systems (SK5) 

• Providing opportunities for students to see connections among science disciplines and between science 
and other disciplines (SK6) 
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• Developing fundamental understanding of chemistry concepts (SK7) 

Scientific Methods (SM) 

• Making students understand scientific inquiry as a process (SMI) 

• Improving students’ ability to think critically about chemistry-related problems (SM2) 

• Focusing upon student experiences with process skills such as inferring, identifying variables, etc. 

(SM3) 

• Providing students with the ability to form a hypothesis and plan an experiment to test the hypothesis 
(SM4) 

• Enabling students to use scientific methods to solve daily problems (SM5) 

• Emphasizing the methods and processes of scientific investigation (SM6) 

• Providing opportunities for students to discover chemistry concepts through laboratory activities) 

(SM7) 

Societal Issues (SI) 

• Promoting the protection, conservation, and improvement of the environment (SI1) 

• Developing a sense of community from the local to international levels (SI2) 

• Emphasizing environmental concepts and world problems (SI3) 

• Including contemporary social problems and solutions for those problems (S14) 

• Providing students with an understanding of the crucial role of chemistry and technology in our society 
(SI5) 

• Recognizing and emphasizing the moral obligation of chemistry and technology to the individual and 
society (SI6) 

• Presenting chemistry as a value-laden subject that has moral and ethical dimensions (SI7) 

Personal Needs (PN) 

Developing the intellect of students (PN 1) 

To contribute to the creation of a favorable environment that will promote the development of students 
as human beings (PN2) 

Providing information and training that will be useful for the students’ future employment (PN3) 
Providing opportunities for the total personal development of the students (PN4) 

Career Awareness (CA) 

Preparing students for chemistry-related careers (CA1) 

Emphasizing the importance of chemistry-related careers to the advancement of science and technology 
(CA2) 

Informing students of the demand for chemists and chemistry teachers (CA3) 

Being geared to the training of future scientists (CA4) 

Promoting awareness of chemistry-related careers (CA5) 

Step 2: The formation of the AHP model for evaluating the effectiveness of the chemistry education based on the 
goals. 

In AHP technique, after the factors and sub-factors constituting the problem are determined, it is necessary to 
form a model to show the relationships of the factors to one another (Saaty, 1980). Therefore, in this study too, 
the goals and sub-goals, which were hierarchically categorized in the first step, were converted into an AHP 
model. As seen in Figure 1, the AHP model for the effectiveness of chemistry education based on the goals 
consists of three levels. The first level includes the objective function. In the present study, the objective of the 
model is “to evaluate the effectiveness of chemistry education based on the goals of chemistry education”. The 
second level contains the main goals. There are five main goals in the model: scientific knowledge, scientific 
methods, societal issues, personal needs, and career awareness. The third level of the model consists of sub-goals 
clustered under the main goals. As seen in the third level of the model, there are 30 sub-goals in total. 
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Figure 1. AHP model for the goals of chemistry education 


Step 3: To make pairwise comparisons of the matrix of each of main goals and sub- goals with a 1-9 scale, and to 
calculate the local weights and the consistency ratio (CR). 

This step consists of the formation of pairwise comparison matrixes of main goals and sub-goals included in the 
proposed model, evaluation of each matrix via 1-9 scale, calculation of local weights, and determination of 
consistency ratio. 

There are six matrixes that must be formed according to the AHP model of this study (Figure 1). The first one is 
the matrix in 5x5 size that covers five main goals existing in the second level of the model. As seen in Figure 1, 
the components of the matrix are scientific knowledge, scientific methods, societal issues, personal needs, and 
career awareness. The other five matrixes consist of sub-goals existing in the third level of the model. The first 
one is the matrix in the 7x7 size that covers seven sub-goals included in the scientific knowledge main goals. 
The second matrix is the one consisting of sub-goals covered by scientific methods. The scientific methods 
matrix is also a 7x7 size matrix. The third matrix consists of sub-goals included by the societal issues main goal, 
and this matrix has a 7x7 size. The fourth matrix in the third level of the model is associated with sub-goals 
covered by the personal needs main goal. As is seen in the model, the personal needs matrix has a 4x4 size. The 
last matrix in the third level of the model is a 5x5 matrix related to the career awareness main goal. 

After the matrixes included by the model have been determined according to AHP technique, it is necessary to 
make pairwise comparisons. In other words, the elements making up the matrixes are to be compared with one 
another. Pairwise comparison is made according to a 1-9 scale (Table 1). In AHP, pairwise comparisons are made 
by relevant specialists who know the nature of the research problem. In the present study, pairwise comparisons 
were made by an expert team consisting of three teachers giving chemistry courses for the 9th grade in 
secondary education schools. The goals of chemistry education were evaluated based on the opinions of the 
expert team according to 1-9 scale (Table 1), and numerical values corresponding to the evaluation findings were 


































www.ccsenet.org/ies 


International Education Studies 


Vol. 5, No. 5; 2012 


assigned to relevant matrixes. 

In AHP, pairwise comparisons are respectively made between two elements of the matrix. In this study, goals and 
sub-goals were indicated as elements. Accordingly, comparison of i. element located in the line of the matrix and 
j. element located in the column of the matrix is made according to Table 1 and the result (ay) is expressed as 
numerical value. On the other hand, the comparison between j. element located in the column of the matrix and i. 
element located in the line is equal to the inverse (ajj= 1/aO of ay. In the light of the above-mentioned information, 
firstly, a pairwise comparison matrix associated with the main goals was formed (Table 3). Pairwise comparisons 
of main goals matrix (Table 3) were made by an expert team consisting of experienced teachers. These pairwise 
comparisons were as follows: How important the goals were in comparison to one another in the effectiveness of 
the chemistry education was determined according to Table 1. For example, how important is the scientific 
knowledge (SK) main goal in comparison to the scientific methods (SM) main goal when the scientific 
knowledge (SK) main goal and the scientific methods(SM) main goal in the matrix (Table 3) are compared? As 
in this study, if the answer is “weakly” more important, the numerical value to be assigned to matrix is 3 
according to Table 1. As the comparison of j. element with i. element is equal to aj, = 1/a; if the comparison of i. 
element and j. element is equal to aij according to the AHP technique, pairwise comparison value related to the 
comparison of the scientific methods (SM) main goal in the column with the scientific knowledge (SK) main 
goal is 1/3. Other elements of matrix were compared in a similar way. Accordingly, pairwise comparisons, local 
weights and consistency ratio (CR) related to the main goals are given in Table 3. 


Table 3. The pairwise comparison matrix and local weights of the main goals 


Main goals 

SK 

SM 

SI 

PN 

CA 

Local weights 

SK 

1 

1 

3 

3 

2 

0.312 

SM 

1 

1 

3 

2 

3 

0.308 

SI 

1/3 

1/3 

1 

3 

2 

0.171 

PN 

1/3 

1/2 

1/3 

1 

2 

0.116 

CA 

1/2 

1/3 

1/2 

1/2 

1 

0.093 

CR: 0.07 


In the present study, local weights and consistency ratios were calculated via Expert Choice (2000). When Table 
3 is examined, pairwise comparisons of the main goals can be seen. In addition, the local weights of each main 
goal can be seen in the last column. Local weight indicates the relative importance of each main goal. In other 
words, it indicates the level of each goal within the whole goal. The total of local weights of goals is equal to 1. 
For the present study, the relative importance of scientific knowledge, scientific methods, societal issues, 
personal needs, and career awareness are respectively 0.312, 0.308, 0.171, 0.116, and 0.093. The consistency 
ratio of the matrix was found as 0.07. This value indicates that the pairwise comparisons are consistent according 
to the threshold values anticipated by AHP assumptions. 

In the present study, after the pairwise comparison matrix related to main goals was formed and local weights 
and consistency ratios were set, pairwise comparisons of the goals in the third level of the model were made. 

Table 4 involves pairwise comparison matrixes and local weights respectively; these are related to scientific 
knowledge, scientific methods, societal issues, personal needs, and career awareness sub-goals. A pairwise 
comparison of scientific knowledge sub-goals was made by the team of experts as follows: How important are 
sub-goals in comparison to one another in terms of scientific knowledge? For example, when reflecting the 
nature and structure of chemistry (SKI) the sub-goal was compared with the “presenting chemistry as a body of 
scientific knowledge” (SK2) sub-goal, the SKI sub-goal was not found to be more important than SK2. However, 
when the SK2 sub-goal was compared with SKI, SK2 was found “weakly” more important than SKI. Therefore, 
while the value related to the comparison of SKI with SK2 is 1/3, the value obtained at the end of comparison of 
SK2 with SKI is 3. In the last column of Table 4, there are local weights related to sub-goals. For example, the 
local weight of “reflecting the nature and structure of chemistry” (SKI) sub-goal is 0.089. This value indicates 
that, for this study, the relative importance of reflecting the nature and structure of chemistry (SKI) sub-goal 
within scientific knowledge goal is 8.9%. Other local weights can be interpreted in a similar way. It was seen 
that pairwise comparisons related to sub-goals forming the scientific knowledge are also consistent (CR: 0.05). 
Pairwise comparisons and local weights of sub-goals related to scientific methods, societal issues, personal needs, 
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career awareness have been determined as in the above-described main goals and scientific knowledge sub-goals. 
The consistency ratio of the scientific methods (CR: 0.08), societal issues (CR: 0.07), personal needs (CR: 0.04), 
career awareness (CR: 0.04) sub-goals was found within acceptable limits. 


Table 4. The pairwise comparison matrix and local weights for sub-goals 


Scientific knowledge sub-goals 

SKI 

SK2 

SK3 

SK4 

SK5 

SK6 

SK7 

Local weights 

SKI 

1 

1/3 

1/3 

3 

1 

1/3 

1 

0.089 

SK2 

3 

1 

1 

3 

5 

1 

3 

0.236 

SK3 

3 

1 

1 

2 

3 

1 

3 

0.206 

SK4 

1/3 

1/3 

1/2 

1 

1 

1/5 

1 

0.063 

SK5 

1 

1/5 

1/3 

1 

1 

1/5 

3 

0.078 

SK6 

3 

1 

1 

5 

5 

1 

5 

0.269 

SK7 

1 

1/3 

1/3 

1 

1/3 

1/5 

1 

0.059 

Scientific methods sub-goals 

SMI 

SM2 

SM3 

SM4 

SM5 

SM6 

SM7 


SMI 

1 

2 

2 

1/3 

1/3 

3 

2 

0.142 

SM2 

1/3 

1 

3 

1 

1/3 

3 

3 

0.152 

SM3 

1/2 

1/3 

1 

1 

1/3 

5 

2 

0.111 

SM4 

3 

1 

1 

1 

1/3 

3 

3 

0.169 

SM5 

3 

3 

3 

3 

1 

4 

3 

0.314 

SM6 

1/3 

1/3 

1/5 

1/3 

1/4 

1 

1/3 

0.042 

SM7 

1/2 

1/3 

1/2 

1/3 

1/3 

3 

1 

0.069 

Societal issues sub-goals 

SI1 

SI2 

SI3 

SI4 

SI5 

SI6 

SI 7 


SI1 

1 

1 

2 

3 

2 

3 

5 

0.242 

SI2 

1 

1 

3 

3 

3 

5 

3 

0.275 

SI3 

1/2 

1/3 

1 

1 

2 

3 

3 

0.130 

SI4 

1/3 

1/5 

1 

1 

3 

5 

5 

0.163 

SI5 

1/2 

1/3 

1/2 

1/5 

1 

3 

2 

0.090 

SI6 

1/3 

1/5 

1/3 

1/5 

1/3 

1 

3 

0.057 

SI7 

1/5 

1/3 

1/3 

1/5 

1/2 

1/3 

1 

0.043 

Personal needs sub-goals 

PN1 

PN2 

PN3 

PN4 





PN1 

1 

3 

2 

3 




0.456 

PN2 

1/3 

1 

1 

3 




0.222 

PN3 

1/2 

1 

1 

2 




0.214 

PN4 

1/3 

1/3 

1/2 

1 




0.107 

Career awareness sub-goals 

CA1 

CA2 

CA3 

CA4 

CA5 




CA1 

1 

3 

3 

1/3 

1/3 



0.163 

CA2 

1/3 

1 

1/3 

1/5 

1/7 



0.049 

CA3 

1/3 

3 

1 

1/3 

1/3 



0.103 

CA4 

3 

5 

3 

1 

1 



0.333 

CA5 

3 

7 

3 

1 

1 



0.352 


Step 4: Calculating the global weights for the sub-goals of chemistry education, and determining the 
effectiveness of chemistry education. 

Operations performed in this step are given in Table 6. As may be seen in Table 6, there are five columns: goals 
of chemistry education, global weights, levels of realization, value of level, and level of effectiveness by 
sub-goals. In this step, firstly, the global weights of sub-goals in the third level of the AHP model (figure 1) were 
calculated by means of Expert Choice (2000). Global weights are calculated according to the local weights of the 
main goals and sub-goals. And the calculated global weights of sub-goals are given in the second column of 
Table 6. Global weight indicates the relative importance of each sub-goal within all of the goals. In the present 
study, global weights demonstrate anticipated or intended realization levels related to the goals of the chemistry 
education. For example, in this study, the global weight of “reflecting the nature and structure of chemistry” 
(SKI) sub-goal was calculated as 0.028. It means that the reflecting the nature and structure of chemistry (SKI) 
sub-goal has a 2.8% relative importance within all of the goals, and the expected realization level is 2.8% when 
all the goals are taken into consideration. 
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Another component determined in this step of the study is the “level of effectiveness” of sub-goals given in the 
last column of Table 6. The levels of effectiveness of sub-goals were calculated by using the global weights and 
evaluation scale in Table 5. Table 5 was used for evaluating the current situation of each sub-goal. This scale was 
adapted from a study in the literature (Yiiksel & Dagdeviren, 2006). The scale consists of six levels. Levels in 
the first column of the scale indicate the realization levels of each sub-goal, and levels in the second column of 
the scale demonstrate the numerical values corresponding to them. The current situation of each sub-goal was 
determined by the expert team by using the scale in Table 5. In this study, the current situation evaluations 
related to sub-goals are given in the third column of Table 6. The numerical value corresponding to the level of 
realization is given in the fourth column of Table 6. 


Table 5. Evaluation scale for goals of chemistry education to current situation 


Levels of realization 

Value of level 

Very good (VG) 

1.0 

Good (G) 

0.8 

Moderate (M) 

0.6 

Negative (N) 

0.4 

Very negative (VN) 

0.2 

Not evaluation (NE) 

0.0 


The level of effectiveness of each goal in the last column of Table 6 was calculated through the multiplication of 
global weight and the value of level. In the last line of Table 6, the total effectiveness of the chemistry education 
is given according to sub-goals. Total effectiveness of the chemistry education consists of the sum of the level of 
effectiveness of each sub-goal. Accordingly, the effectiveness of chemistry education based on the goals of 
chemistry education is 68.96%. This can be interpreted as follows: The goals of secondary education ninth grade 
chemistry education were realized at the level of 68.96%. 


Table 6. Calculating the level of the effectiveness of the chemistiy education based on the goals of the chemistry 
education 


Sub-goals 
of chemistry 
education 

Global 

Weights of the 

sub-goals 

(gw) 

Levels 

of realization 
of sub-goals 

Value 

of level 
(vl) 

Level 

of effectiveness 
by sub-goals 
(gw x vl) 

SKI 

0.028 

VG 

1.0 

0.0280 

SK2 

0.074 

VG 

1.0 

0.0740 

SK3 

0.064 

G 

0.8 

0.0512 

SK4 

0.020 

G 

0.8 

0.0160 

SK5 

0.025 

G 

1.0 

0.0250 

SK6 

0.084 

VG 

1.0 

0.0840 

SK7 

0.019 

VG 

1.0 

0.0190 

SMI 

0.044 

M 

0.6 

0.0264 

SM2 

0.047 

M 

0.6 

0.0282 

SM3 

0.034 

G 

0.8 

0.0272 

SM4 

0.052 

M 

0.6 

0.0312 

SM5 

0.097 

M 

0.6 

0.0582 

SM6 

0.013 

M 

0.6 

0.0078 

SM7 

0.021 

M 

0.6 

0.0126 

SI1 

0.041 

M 

0.6 

0.0246 

SI2 

0.047 

VN 

0.2 

0.0094 

SI3 

0.022 

N 

0.4 

0.0088 

SI4 

0.028 

VN 

0.2 

0.0056 

SI5 

0.015 

M 

0.6 

0.0090 
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SI6 

0.010 

N 

0.4 

0.0040 

SI 7 

0.007 

N 

0.4 

0.0028 

PN1 

0.053 

G 

0.8 

0.0424 

PN2 

0.026 

M 

0.6 

0.0156 

PN3 

0.025 

M 

0.6 

0.0150 

PN4 

0.012 

M 

0.6 

0.0072 

CA1 

0.015 

M 

0.6 

0.0090 

CA2 

0.005 

M 

0.6 

0.0030 

CA3 

0.010 

M 

0.6 

0.0060 

CA4 

0.031 

M 

0.6 

0.0186 

CA5 

0.033 

M 

0.6 

0.0198 


Total effectiveness of the chemistry education according to the sub-goals 0.6896 


4. Conclusion 

In this study, an attempt was made to develop a method of measurement and evaluation that overcomes the 
difficulties encountered within the process of determining the effectiveness of the chemistiy education based on 
the goals of chemistry education. In the proposed method, Analytic Hierarchy Process (AHP), which is a 
multi-criteria decision technique, developed by Saaty (1980) was used. 

In the present study, a large number of the goals of chemistry education were hierarchically grouped according to 
their common features from a holistic perspective by means of the proposed method. Thirty chemistry goals 
taken as a basis in the study were converted into an AHP model in five main groups. The constituted AHP model 
is considered important in terms of determining the position of each goal within the whole. That is because the 
separate evaluation of the effectiveness of chemistry education for each goal will not allow for an evaluation 
related to the whole. Just the level at which the goal taken into consideration is realized can be determined. 

Another advantage of the proposed method is the possibility of determining the local weight of each goal in the 
AHP model constituted from a holistic perspective. Local weights indicate the relative importance levels of goals 
within the group they exist in. As is stated in the introduction section, it is not possible to say that the goals of 
chemistry education have the same importance levels. Therefore, it is thought crucial to determine the level at 
which each goal is important for chemistry education. 

The relative importance level of each qualitative goal can be calculated by means of the AHP technique used in 
the study. As a matter of fact, one of the advantages of the AHP technique is that it can model factors with both 
qualitative and quantitative structures. In addition, the consistency or inconsistency of pairwise comparisons 
allowing for the calculation of relative importance can also be calculated by means of AHP technique. This 
characteristic of AHP allows for an awareness of the presence or absence of any inconsistency in the 
effectiveness research. As a matter of fact, inconsistencies are to be expected in evaluations made in accordance 
with opinions of human factor. By means of the AHP technique used in this study, the consistency levels of 
pairwise comparisons were determined, and they were found to be within acceptable threshold values. 

In the study, the global weights of each goal were determined via AHP technique. The determination of global 
weights answers the question as to what is of relative importance in each goal within all of the goals. For 
example, it is not possible to say that the “reflecting the nature and structure of chemistry” and the “enabling 
students to use the scientific method to solve daily problems” goals can have the same relative importance in 
chemistry education. Accordingly, it is necessary to consider the differences between the relative importance of 
goals during the evaluation. As a matter of fact, the findings of this study indicate that the relative importance of 
“reflecting the nature and structure of chemistry” goal within all of the goals is 0.028, and the relative 
importance of “enabling students to use the scientific method to solve daily problems” goal within all of the 
goals is 0.097. On the contrary, the separate evaluation of the effectiveness of chemistry education for each goal 
does not provide an opportunity to make an evaluation of the whole. However, the relative importance of each 
goal may vary. 

In this study, the effectiveness of chemistry education based on goals was determined by using the advantages 
provided by the AHP technique. In the present study, the expected theoretical value range of the effectiveness is 
0 - 1. The nearness of effectiveness value to 1 means the perfect realization level of goals, and the nearness of 
effectiveness value to 0 means an insufficient realization level of goals. The effectiveness level was calculated as 
0.6896 in this study. In other words, 31.04% of the goals anticipated in the chemistry education could not be 
realized. This value shows that there is not a perfect effectiveness level in terms of the grade where the study was 
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conducted. The acceptability of the calculated effectiveness level may vary according to the judgments of the 
decision-makers and the levels set. Just as the effectiveness of chemistry education can be determined, based on 
all of the goals by means of the method proposed in the study; the level of effectiveness in terms of each goal 
can also be calculated by this method. For example, it was found in the present study that 31.04% of goals could 
not be realized. The goals making up the level (31.04%) that could not be realized can be determined in detail by 
the global weights of the sub-goals and the level of effectiveness by sub-goals values. The global weights of the 
sub-goals calculated by the proposed method indicate the expected realization level of each goal and the level of 
effectiveness by sub-goals values show the realization level of goals (Table 6). Accordingly, the difference 
between the level of effectiveness by sub-goals and the global weights of the sub-goals demonstrate the 
unrealized part of each goal. 

It is possible to say that the criticisms of other measurement and evaluation techniques or methods in the 
literature (Gelbal & Kelecioglu, 2007; (j’elikkaya et al., 2010; Okur & Azar, 2011) can also be made of this study. 
Accordingly, it is not possible to say that there will be no criticism of the assumptions, content and 
implementation of the proposed method. The first criticism may be of the validation of the goals of chemistry 
education taken as a basis for the present study. As is expressed in the methods section of the study, in this study, 
the effectiveness level of chemistry education was determined based on the classification made by Gayon (2010). 
There are two main reasons for the use of this classification: The first one stems from the purpose of the study. 
That is to say, it was intended to develop a method for determining the effectiveness of chemistry education. The 
issue of what the goals of chemistry education should be or the content validation of the goals used in the study 
does not constitute the problem of the study. In this study, the chemistry education goals and classification as set 
by Gayon (2010) were assumed to be acceptable. The main reason for this assumption was that the classification 
made by Gayon (2010) concerning the goals of chemistry education showed a similarity with the goals included 
in the curriculum of the school where this study was conducted. Despite these reasons, there might be objections 
to the main goals and classification taken as basis in this study. However, that does not affect the operation of the 
proposed method. 

Although the method proposed in this study intends to determine the effectiveness of chemistry education, it has 
a structure which can be implemented in other disciplines as well. The aspect which needs to be modified is the 
constitution of an AHP model according to the number and classification of goals to be taken as the basis in the 
study. There is no difference in other operations of the method. 

In future studies, the issue of effectiveness could be examined through a fuzzy logic approach. The main reason 
for the use of such an approach is that some of the goals of education may be vaguely stated (Baykul, 1992), 
education services are often intangible, and accordingly direct measurement is difficult (Tsinidou et al., 2010). 
Therefore, in future research, the local and global weights of chemistry education goals can be calculated by 
means of fuzzy numbers. Moreover, overall evaluations can be made on the subject of effectiveness by using 
data that represent the views of students as well as those of course or school directors. 
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